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‘ V",;.-^21 May 1975 fm* ’ • 


IjTG Samuel V. Wilsoii ^ ! 

Deputy to the DCI for the \ 

Intelligence Community - 

Washington, D. C. 20505 ' ' 

' Dear LTG Wilson: . ‘ 

" : v -New Business for 20 May Meeting of the NSCIC Working Group 

' r : * — *• [■' ■ * 

■ Ambassador Ellsworth requested that I forward the following 

comments and suggestions regarding the 20 May meeting of the NSCIC 
Working Group, and possible new business for future meetings. 

1 ^ * .Status of C ommittees on Con sumer Evaluation and Consumer 

' Guidance • ” . ' 1 ■ . ] ’ ~ 


These committees could be highly useful. Hie committee for 
‘ '’Consumer Evaluation^ Current Intelligence Products'’ represents an 
. Important step forward in developing the kind of detailed consumer evaluation 
! of the intelligence product which is essential to support the NSCIC, and to 
‘ provide the Intelligence Community with a user view of its output. Such a 
committee could not only .establish suitable procedures for obtaining U6er 
views f i om different agencies, but serve as a mechanism for evaluating • 

• major products. The "Consumer Guidance" committee also seems useful, 
and could serve a valuable purpose in giving the Intelligence Community a 
better view of the needs of different agencies for given intelligence products, 
;:;-n and for substantive detail in providing intelligence on given subjects. This 
; committee might be particularly useful in reviewing the scheduling of major 
intelligence products relative to the ICXQs, Objectives, and needs of given 
consumers. i 

I ' !• ; s 

■ ' • • ’ • i ' ' . . 

■» ^ proposal: for setting up such committees was originally 

scheduled to be distributed to members of the NSCIC Working Group by ■ j 
15 April. It would be bolpful to have such a proposal as soon as possible ! 
so that it can be discussed at the next meeting o£the Working Group. • I ' 
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2. Date of the Next Meetim 


It would be helpful if the next meeting of the NSCIC/WG could . 

1 bo timed so that members had an opportunity to review NIAM 35/36 before 
.the meeting. We would, like to discuss this NIAM, and to present a draft 

I "critique" as requested in your note of 11 March, 1975. 

h . ; •. . . . / ••• - • 

3« Need for An : Annotated Bibliography of NIEs, SNIEs. NIAMs, 
and NSSM Support Products 

It became apparent in surveying various components of OSD that 
national intelligence products are not being properly utilized because no 
adequate bibliography exists of such documents. We have looked Into this 
problem, and the situation can be summarized as follows: 

* * 

•t ' : ‘ !", I ’ 

-- We have surveyed OSD views towards user review of NIEs, 
v ‘SNIEs, and N JAMs. We strongly endorse this idea. We have selected NIAM 
35/36 for inital review and we have determined other documents for future 
•review. Wo are developing methodology for user review. 


’"•''I' We found in doing this survey that we have a major problem 

' - in OSD because no useful or timely annotated bibliography exists of NIEs, 
b/i' NIAMs, and SNIEs, The overall impact of the national intelligence effort 
. .is sharply reduced by poor information flow caused by lack of a timely 
" watt to survey overall intelligence effort, a lack of staff knowledge of 

national level products, and a lack of any way to learn what has been done 
: ; in the past, „ 

-- The DJA USIB library tried to create an index of NIEs, SNIEs, 
and NIAMs for us. We found its holdings had serious gaps; and were some- 
times out of date. 

-» We found some products not getting to. key staffs in PA&E, 

ISA (P/P) and DDR&E because they only need these for specific studies 
and have no timely or comprehensive index to consult. 


• " There used to be a comprehensive index of NIEs, SNIEs, and 

NIAMs until |l.964. It was not annotated and was not organized into functional 
categories, 

-- There is now only an annual index of NIEs, SNJEs, and NIAMs 
i' which comes out only three months after end of year surveyed. It does not 
sfiow previous products not updated during that year ( USIB-D-17. 1,5, Estimates 
published during 1974, 5 March 1 975) . 

VvV- ' SECR^ ^OPORN 
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' . — DIA doee have an excellent semi-annual and annotated index 

of DIEe, SDIEs, DIAMs, and DIEMb (DIA, "Estimat es Publications List,'* 
SU2008/DE-X (S)} published in last six months. It also lists all documents 
published in last five years* j 


This lack of a proper and timely annotated bibliography of national 
intelligence documents should be corx'ectcd as soon as possible. Accord- 

we request that tJie IC Staff take the following steps to solve the 
problem: 

t \ ".-'i .... 

■"* Immediately add an attachment ot the DCI "Status of Work 
in Progress Report” which lists all products’ produced since the period 
covered in USIB-D-17, 15* 


— Prepare a sample annotated bibliography of the type developed 
by DIA in S-2008/DE~X{S) for the period since USIB-D-17. 15 and present 
this at the next meeting of the NSCIC/WG. t 

■'V'.!;:,.; Begin development of a comprehensive annotated bibliography 

Of the national intelligence product. This should include all documents not 
updated in the last five years. . It should include a future production schedule 
, > for the comining 12 months. It should be updated semi-annually, and it ■ 

! should have a- specific listing o£ all regularly scheduled products hot updated 
! during the last year. 

, . • 1 . • ■ . K - • ' ' 

This document will not only.be a major step forward in solving 
the information flow problem, but provide the NSCIC and NSCIC/WG with 
« an overview of the national product effort. 



OSD is developing a package of user comments on the FY 76 
KIQs. The main result of surveying users has, however, been to indicate 
that many do not feel that the KIQs are important enough to merit detailed 
review. The FY 75 KIQs are felt to be so general that they do not provide 
meaningful guidance, further, the KIQ strategy reports suffer from the 
>same problem. They do: not address what substantive issues or data needs 
should be addressed in answering the KIQ, and they do not evaluate whether 
the products listed actually answer the KIQ. Instead, they associate all 
related output to a given KIQ, and make. the volume of related output the 
measure of success. 
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DIA is providing a coordinated DoD input for the FY 76 KIQa 
in the form of Defense Key Intelligence Questions. Unlike the present 
KIQs, those provide (a) a rationale for asking the question, (b) a detailed 
listing of the substantive data needed to answer the question in the form 
of "Sub-DKIQs, " and (c): an indication of priorities. Such a format is 
distinctly preferable to that of the FY 75 KIQs. It is particularly important 
that the FY 76 KIQs list: in detail the substantive areas where improved 
V intelligence is required'to answer the question. Accordingly, it would be 
useful if the IC, Staff would examine the format of the DKIQ s as soon as 
they are submitted by DIA, and present. ideas for improving the format 
of the KIQs at the next meeting' of the NSCIC/WG. , 

It would also he useful if the IC Staff could present a staff 
paper with an evaluation of the effectiveness of the KIQ system., and a 
clearer explanation of how the KIQs are derived. This should contain the 
following information: 

A. | ■ .# 

-- The views of the individual NIOs and a sampling of key 

users to soo how valuable they felt the FY 75 KIQs wore as guidance to 
the community, •. ■ . : 

- - A detailed analysis of how many NIEs,. NIAIVIb, SNIEs 

■ generated since the issuing of FY 75 KIQs were specifically in response 
to KIQs or were in response to other forme of specific tasking. Do the 
! KIQs really guide the community? . • 

! • v * 1 * • « 

—A description of how various agency inputs are combined 
* into KIQs, of who choses which inputs are made into KIQs, and of how 

those choices are made. I 

. c • . • ■ ■ i ‘ « ■ ■ 

5. The KIQ FY;75 Evaluation Performance Repor t 

It is not clear that the present structure of the KEP is what is 
required, or that the effort involved id justified. The detailed formats 
included in the KEP questionaire may be of uncertain value, and could, 
result in the KEP effort being little more than a "paper chase." It would 
be useful if the IC Staff - could prepare an analysis of the KEP process which 
deals with the following issues. 

— Which dtaffs used the I<IQs? Were the FY 75 KIQs adequate? 

m . ... What kind of user review of KIQ performance would be most 

useful? Should the NIOs be tasked with surveying users in filling out an 
improved KEP? 
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m- Should tho NIOs bo involve din this kind of resource 
analysis related activity? 1 

■■ j . . . . ! v • ' 

-- Is an updated KTQ Strategy Report really all the review 

effort required? Is the KEP worth its cost? 

Are the’KIQs specific and detailed enough to justify and 
permit detailed audit trails of the type the KEP system calls for? 

-- Does the KEP system for relating levels of effort* costs, 
and collection means to: specific KIQs produce meaningful data? 

-- Would sampling techniques, including product evaluation 
and user reactions, produce more valuable results? 

' 6, The M Family, of Intelligence Documents 11 


• It may be helpful to clarify in more detail the comments Mr, j 

Ellsworth made on this document. The present draft should be expanded to: 

• , l . • ; 

mm- Describe who writes the document, how it is written, whoso 
views and Inputs are solicited, and how user views ai’e surveyed and taken 
into account. •’ ,= ■_ . ... . \ 

; Describe the planning cycle involved, and how it relates 

to the budget cycle and impacts bn intelligence resource requirements, 

I . mm Describe how the docuinents are used in the Intelligence 

Community, their impact, and who they influence, . *• 

; ; ; ■ . . ■. j .!• . . ■ ■ !• 

It would also be helpful if the I.C Staff could survey user* reactions 
to the value and success of tho current documents, and if tho informal 
comments of the individual NIOs on the value and success of each document 
could be surveyed and presented to members of the Working Group in the 
near future, . ' : 


7. Tho Probl 
Estimate: 


Impressi ng tTncer ta int y in In te lligence 


The importance of this IC Staff paper cannot be over-empha sized, 
and it must not be generated in the form of a "broad brush" treatment, or 
g^oss over the problems involved. A brief review of national intelligence 
products indicates that farr too many* 
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Fail to explicitly state the limitations in the intelligence 
available or the methods of analysis used and present them in summary 
form for quick review. Jh many cases, 5 even those limitations that are 
listed are buried in the text or listed only in footnotes or annexes. 

.'V * % \ 

J* -- Fail to explicitly discribe the methodology used when hard 

. intelligence data ie lacking, j 

■ ' I ►’ | f • . ; 

• ,Fail to clearly distinguish reasonable c one liis ions or logical 
yiewa from conclusions based on intelligence. 

- - Fail to describe the uncertainty present in given data, 
methods of analysis, or., conclusions, 

i 

' ; ; v -» Fail to quantify uncertainty, fail to describe the method 

used to quantify uncertainty, or fail' to show a range of numbers. 

' • «•» Fail to list the explicit intelligence data on. which a con- 

clusion is based, and provide only a broad or vague rationale, j. 

. i • ■ • 

The IC Staff paper should specifically address these problems, , 
It should also survey the last year's output, and specifically list those docu4 
ments where it is felt such problems did not occur. It should not simply ! 
survey the methods being used in the Community. It should specifically 
identify the specific products to which they, were applied, and it should j 
indicate the percentage , of total annual output in which such methodologies 
were .applied. ; ... 


Anthony H, Cordeeman 

Chief, Product Evaluation Division 

OASD (I)/IM 

E^t, 0X7-4436 

i(Foi’ Ambassador Ellsworth) 
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